Module 9 Lecture - Repeated Measures ANOVA

Analysis of Variance

Quinton Quagliano, M.S., C.S.P

Department of Educational Psychology

1 Overview and Introduction

Agenda

1 Overview and Introduction

2 Null and Alternative Hypotheses

3 Outcomes and the Type I and Type II Errors

4 Probability Distribution Needed for Hypothesis Testing

5 Rare Events, the Sample, Decision, and Conclusion

6 Conclusion

1.1 Textbook Learning Objectives

  • Differentiate between Type I and Type II Errors
  • Describe hypothesis testing in general and in practice
  • Conduct and interpret hypothesis tests for a single population mean, population standard deviation known.
  • Conduct and interpret hypothesis tests for a single population mean, population standard deviation unknown.
  • Conduct and interpret hypothesis tests for a single population proportion.

1.2 Instructor Learning Objectives

  • Be able to identify different parts of a hypothesis given a narrative example
  • Understand the connection between \(\alpha\), p-values, probability, and Type I and II errors
  • Be able to work through practical inferential statistics computations for a single sample scenario

1.3 Introduction

  • In statistics, hypothesis testing is the process by which we evaluate whether data supports making a certain conclusion beyond a reasonable doubt

  • We have certain steps to work through a hypothesis test:

    • Set up the contradictory null and alternative hypotheses
    • Collect data in a sample
    • Determine an appropriate inferential test and distribution to represent our variables
    • Conclude whether we can reject the null hypothesis or if we cannot
  • Important: I think it's important to emphasize that it is not our 'goal' to reject the null hypothesis - as empiricists, our orientation should be towards truth, not a certain outcome

2 Null and Alternative Hypotheses

Agenda

1 Overview and Introduction

2 Null and Alternative Hypotheses

3 Outcomes and the Type I and Type II Errors

4 Probability Distribution Needed for Hypothesis Testing

5 Rare Events, the Sample, Decision, and Conclusion

6 Conclusion

2.1 Introduction

  • A hypothesis is just a prediction or a statement of a possible outcome
    • E.g., I hypothesize (or predict) that college students who are Gen Z have significantly worse attention than Millennial college students
    • Most scientific studies begin with some hypothesis of what outcomes will look like
  • Discuss: Consider that you are a middle-school teacher being asked by administration to implement an especially permissive attendance structure, write a hypothesis of what this will do to student math scores
  • When we make hypotheses in statistics, we do so in a contradictory pair by making a null hypothesis and an alternative hypothesis
    • In frequentist statistical testing, we frame our inferential tests as helping us determine whether to reject the null - there is a whole other field of statistics called Bayesian statistics that has a somewhat different focus
  • Important: It's important to understand that, as part of the statistical testing process, we should always consider what our null and alternative hypothesis is!

2.2 Null Hypotheses

  • A null hypothesis is a prediction or statement of no difference between two or more things
    • It is usually written as \(H_0\)
    • Opposite of alternative hypothesis
  • A null hypothesis is often written to include an equal sign such as with:
    • \(=\)
    • But also, \(\geq\) & \(\leq\)
  • Example: There is no difference in attention between Gen Z and Millennial college students
    • Represented in notation \(H_0: Attention_{Z} = Attention_{Millennial}\)
  • Discuss: Write the null hypothesis for comparing mental health between women and men in the legal field, where they have equal levels of depression

2.3 Alternative Hypotheses

  • Our alternative hypothesis suggests a difference between two or more things
    • It is most often represented as \(H_A\) or \(H_1\)
    • Opposite of Null Hypothesis
    • This is usually close to how we would phrase a study hypothesis
  • The alternative hypothesis does not contain equal signs
    • This would include \(\neq\), \(>\), and \(<\)
  • Example: There is a difference between Gen Z and Millennial college students in attention
    • Represented in notation \(H_A: Attention_{Z} \neq Attention_{Millennial}\)
  • Discuss: Write the alternative hypothesis for comparing mental health between women and men in the legal field, where women have higher levels of depression

3 Outcomes and the Type I and Type II Errors

Agenda

1 Overview and Introduction

2 Null and Alternative Hypotheses

3 Outcomes and the Type I and Type II Errors

4 Probability Distribution Needed for Hypothesis Testing

5 Rare Events, the Sample, Decision, and Conclusion

6 Conclusion

3.1 Decision-making

  • When we gather and analyze the data we gather from a sample, we use those results to make a decision
    • The decision comes down to whether we can reject the null hypothesis or not (sometimes called ‘retaining’ the null hypothesis)
    • Ideally we avoid making a type I or type II error with our decision
Reality (Truth) Decision: Reject H₀ Decision: Fail to Reject H₀
H₀ is True Type I Error (α) False positive Correct Decision (True negative)
H₁ is True Correct Decision (True positive) Type II Error (β) False negative

3.2 Types of Errors

  • A type I error is when we decide to reject the null hypothesis, when it is actually true
    • The probability of a Type I error occurring (i.e., \(P(TypeI)\)) is given as \(\alpha\)
  • Discuss: Review: where did we come across alpha before? Explain it's significance in the context of module 8!
  • A type II error is when we decide to retain the null hypothesis when it is actually false
    • The probability of a Type II error occurring (i.e., \(P(TypeII)\)) is given as \(\beta\)
  • Important: When I was first trained on statistics, I was taught to view the Type I error as being more 'egregious' - 'the number one error to avoid!' - but realistically, they both result in flawed conclusions which could be dangerous
  • Ideally, both \(\alpha\) and \(\beta\) should be as small as possible, as to try an avoid making an error of any sort!
    • An extension of this is that power, the likelihood we correctly reject the null when it is false
    • \(Power = 1 - \beta\)
  • Question: In a hypothetical experiment when comparing students before and after a new instructional style, I correctly decide I can reject the null hypothesis that there was no change. This is accurate, as there actually was a change. What error occurred here?
    • A) No
    • B) error
    • C) []{.quarto-shortcode__-param
    • D) "

4 Probability Distribution Needed for Hypothesis Testing

Agenda

1 Overview and Introduction

2 Null and Alternative Hypotheses

3 Outcomes and the Type I and Type II Errors

4 Probability Distribution Needed for Hypothesis Testing

5 Rare Events, the Sample, Decision, and Conclusion

6 Conclusion

4.1 Introduction

  • Its important that we identify the sampling probability distribution we can use to represent a certain variable and it’s mean or proportion
    • Effectively, this is where we are choosing a test - a very important step in our analysis!
  • Important: Choosing the 'right' test for certain situations is probably the most valuable skill a statistician has!
  • Discuss: Review: What is the formula for the SE of the mean and how does that tie back to the normal distribution of sample means?

4.2 Assumptions

  • An assumption is some prerequisite of the distribution we are using

  • Take for example: the z-test for a population mean with the normal distribution. We need:

    • \(\mu\), \(\sigma\), and the population must be normally distributed / large enough to appeal to the CLT
    • We must also have a simple random sample from the population
  • Discuss: Review: Describe why a simple random sample is often impractical. This is where research design intersects with analysis!
  • The t-test for a population mean with the t-distribution assumes we also have a simple random sample from a normally distributed population
  • Important: Different inferential tests have different assumptions, and violating those assumptions can have big consequences! Too many researchers outright ignore these for convenience

5 Rare Events, the Sample, Decision, and Conclusion

Agenda

1 Overview and Introduction

2 Null and Alternative Hypotheses

3 Outcomes and the Type I and Type II Errors

4 Probability Distribution Needed for Hypothesis Testing

5 Rare Events, the Sample, Decision, and Conclusion

6 Conclusion

5.1 Introduction

  • While accounting for the information, statistics, parameters, and sampling distribution we have do help us make good choices about hypothesis testing - they don’t tell us everything
    • We always have to remember that things in statistics are probabilistic!
    • With our sampling and data gathering we may run into a rare event
  • To account for the possibility of a rare event, we test the null hypothesis somehow
  • Discuss: Try re-explaining, in your own words, what 'probablistic' means and why it's readily applicable to the practice of statistics

5.2 Using the Sample to Test the Null Hypothesis

  • This is where we introduce the p-value, which is the chance that, under the null hypothesis being true, our results will be as extreme as they are
    • Example: a test result returns a p-value of 0.07. Assuming the null hypothesis is true, this result only had a 7% chance of occurring.
  • Effectively, when we test against the null hypothesis and determine a p-value, we are trying to gauge how likely our results were to occur if the null hypothesis was true
    • If our results are especially rare under the null hypothesis (i.e., a low p-value), then we may be inclined to believe that our case is somehow truly different, and thus the null hypothesis is incorrect and can be rejected
  • Important: There are many misunderstandings people have about statistics, but failing to understand the meaning of p-values is probably the single most common and pervasive errors people make in interpreting statistics.

5.3 Decision and Conclusion

  • How do we determine if the p-value is, “rare enough”?
    • Ideally, we test it against a preset/preconceived significance level, also given as \(\alpha\).
    • We decide whether our p-value is \(\geq\) or \(<\) our \(\alpha\)
  • Important: This is the exact same alpha that showed up in regard to confidence level in the last module and earlier in our Type I error! We *choose* what we wish our significance level to be at the onset of our analyses.
  • If you don’t see further information, the most common significance level is \(\alpha = 0.05\)
    • However, this isn’t a hard set rule.
  • Discuss: Take a look at the prior discussion on Type I and Type II error - try using those terms to discuss why have too high of an alpha level might be risky

5.4 Tails of a Test

  • A test may be described as two-tailed, left-tailed, or right-tailed, dependent on the sign used in the alternative hypothesis
    • \(H_a: P > 0.5 \rightarrow\) right-tailed (one-tailed)
    • \(H_a: P < 0.5 \rightarrow\) left-tailed (one-tailed)
    • \(H_a: P \neq 0.5 \rightarrow\) two-tailed
  • This needs to be set as part of the study set up, not during analysis!
  • Discuss: Consider the following example: Johnny predicts that Samantha has more money than Becky right now. Is this a one-tailed or two-tailed test and why?

6 Conclusion

Agenda

1 Overview and Introduction

2 Null and Alternative Hypotheses

3 Outcomes and the Type I and Type II Errors

4 Probability Distribution Needed for Hypothesis Testing

5 Rare Events, the Sample, Decision, and Conclusion

6 Conclusion

6.1 Recap

  • Hypothesis testing is the backbone of inferential statistics, and testing against the null hypothesis is how we determine whether our results were just a chance “fluke” or likely some real difference!

  • Proper hypothesis testing hinges upon wise identification of what values we have and what distribution best describes our variables. Each distribution and test we could use has a set of assumptions we need to be aware of

  • P-values, alpha, beta, and the two hypotheses are all connected, and we must be careful to make the correct decisions - wary of the relative risk of Type I and Type II errors

6.2 Lecture Check-in

  • Make sure to complete any lecture check-in tasks associated with this lecture!

Module 9 Lecture - Repeated Measures ANOVA || Analysis of Variance